Repository navigation
feat(libtorch): report device identity per index - #35
NicolasRouquette wants to merge 5 commits into
Conversation
Brings in the upstream PR branch (lean-dojo#35) at 65fd9d9, one commit on upstream main b062b9a: Runtime.Autograd.LibTorch.deviceInfo and currentDeviceInfo read at::cuda::getDeviceProperties(index) and cudaDeviceGetAttribute per index, with DeviceInfo.format for benchmark headers. Successor of the closed lean-dojo#30. The compiled-architecture fields are gone with the custom kernels.
|
Hey Nicolas, thanks! Per-device identity is useful. Could you check this together with #36? The CUDA property queries need CPU-only SDK handling, and the tests should not assume nativeAvailable means a CUDA device exists. Also, the statement in DeviceInfo.lean that TorchLean compiles no device code of its own will need updating when our pending custom-kernel/NVRTC work lands. Please keep the SDK-packaged architecture list separate from runtime-compiled kernels in that explanation. Keeping this open for those integration checks. |
The LibTorch bridge pinned `c10::kCUDA`, so a build without a CUDA device had no backend at all: `unavailable.c` fails every buffer operation, and the device buffer suite could only run on a GPU host. The bridge now selects its device once, at its first call, and runs on the host through ATen's CPU kernels when the SDK has no CUDA support or when `TORCHLEAN_LIBTORCH_DEVICE=cpu` asks for it. Nothing is reimplemented: every ATen operation already goes through `torchlean::options()`. Selection: `TORCHLEAN_LIBTORCH_DEVICE=cpu` selects the host and `=cuda` insists on a CUDA device. Unset, a visible CUDA device is selected when the SDK has CUDA support; otherwise the host is selected only when the SDK has no CUDA support at all, so a CUDA-enabled SDK without a visible device stays `RuntimeStatus.nativeUnavailable` rather than silently running on the host. CMake no longer requires the `torch_cuda` target. A CPU-only SDK compiles the backend with `TORCHLEAN_LIBTORCH_CUDA=0`, without CUDA headers, the CUDA toolkit, NVRTC or the driver API; the CPU-only pip wheel is a complete SDK for it. `sdk.txt` records the choice. Runtime-compiled kernels (the generated operations of `NN.Kernel`) need NVRTC and the driver API, so they compile only with a CUDA-enabled SDK, and `run_kernel` refuses at its entry unless a CUDA device is selected: on the host a generated operation fails through `IO` with an error that names the requirement, before any CUDA call. The CUDA suite runs its custom-computation checks on a CUDA device and checks that refusal on the host. Lean: `Runtime.Autograd.LibTorch.DeviceKind` and `deviceKind` report the selection. `RuntimeStatus` keeps its three states, with `.nativeAvailable` meaning that a device is selected, CUDA or host, so `requireNativeRuntime` and every existing match are unchanged. On the host the native allocator counters and driver memory read zero, `synchronize` and `emptyCache` return at once, `setMemoryFraction` and `setDevice` are rejected, and the suite skips its native memory probes. The suite prints the device and checks that the selection agrees with the visible device count and with the environment variable.
…nd CUDA Programs made of the IEEE basic operations, gathers and lookups give bit-identical results on the two devices; transcendental functions come from different libraries on the host and on CUDA and differ at the ulp level, and reductions differ in order.
`nativeAvailable` covers the host now that the bridge can run there, so `requireNativeRuntime`,
which the suite called for `TORCHLEAN_REQUIRE_CUDA=1`, let a run on a CPU-only SDK or under
`TORCHLEAN_LIBTORCH_DEVICE=cpu` pass a check meant to say that a GPU ran it.
`Buffer.requireCudaDevice` asks for the device kind as well, and the suite uses it for that
variable. `TORCHLEAN_REQUIRE_NATIVE=1` keeps the bridge-only meaning for CPU-only SDK sessions,
so that such a job fails on the unlinked default backend instead of passing with the device suite
skipped.
In the bridge, `require_cuda("<feature>")` is the entry check for an operation with no host
implementation (device selection, the allocator memory fraction and runtime-compiled kernels use
it), so it fails with a message naming the feature rather than with a CUDA call on a CPU tensor. The
READMEs, the installation page and the execution-modes chapter say what each variable rejects, and
quote the new messages.
Configured binary arithmetic computes only in runtime-compiled kernels, which the tape reaches
through pure entry points, where an error ends the process. The encoded upload therefore calls
`require_cuda` too, so on the host such a computation fails through `IO` when its tensor is
uploaded. The host branch of the CUDA suite checks both refusals and runs the custom-autograd checks
that need no generated kernel: the native-precision recordings and the resident-buffer checks.
`LibTorch.deviceInfo index` reads one device's name, compute capability, SM count, clocks, bus width, total memory, and the driver and runtime versions through the SDK's per-device cached properties; `currentDeviceInfo` reads the device `getDevice` selects. Querying by explicit index keeps the answer independent of the selected device, so switching devices cannot report a stale record. Without LibTorch, the queries fail through `IO` like the other controls. `NN/Tests/Runtime/Cuda/DeviceInfo.lean` checks every visible device under its own index, the current-device agreement, index rejection, and, in a fresh process selected by `TORCHLEAN_LIBTORCH_DEVICE_PROBE=switch`, that switching devices changes the reading.
…ty queries The identity exports read `cudaDeviceProp` and `cudaDeviceGetAttribute`, which a CPU-only SDK does not provide, and the test read `nativeAvailable` as "a CUDA device is present", which the host bridge no longer implies. The exports now compile under `TORCHLEAN_LIBTORCH_CUDA`; a CPU-only SDK gets exports that fail through `IO`, as an index outside the visible count does on a CUDA-enabled SDK. A CUDA-enabled SDK still answers while the bridge runs on the host, since the queries describe the visible devices rather than the selected one. `NN/Tests/Runtime/Cuda/DeviceInfo.lean` follows the device the bridge selected: on a CUDA device it runs the readback and the switching probe, which now requires a CUDA device; on the host with a CUDA-enabled SDK it runs the readback alone, because the host rejects `setDevice`; on a CPU-only SDK it checks that every query fails. The module docstring keeps the SDK's packaged architectures, a property of the SDK build, apart from the kernels TorchLean compiles at run time: NVRTC compiles the generated operations of `NN.Kernel` for the compute capability of the device they run on.
65fd9d9 to
cfbcb9a
Compare
|
Rebased onto current
Verification. RTX A4500, pip torch 2.11.0+cu128 with a CUDA 12.8 toolkit, and pip torch All four runs end in |
Stacked on #36
This branch carries #36's three commits first; this PR's own are the last two. #36 gives the
bridge its host device, and this PR's queries must say what they do there, so the two are checked
together. Merging #36 first leaves this PR with its own two commits.
What
Runtime.Autograd.LibTorch.deviceInfo (index : UInt32) : IO DeviceInfoandcurrentDeviceInfo, the LibTorch follow-up to #30: name, compute capability (major * 10 + minor, so86reads assm_86), SM count, core and memory clocks, memory bus width, totalmemory, and the driver and runtime versions, plus
DeviceInfo.peakBandwidthGBsand a one-lineDeviceInfo.formatfor a benchmark header.Buffer.memoryanswers how much device memory thereis; this answers which device.
The review finding from #30
#30 cached one
cudaDevicePropunder a process-widepthread_once, so after switching devicesit kept reporting device 0 and could combine fields from two cards. Here every query takes an
explicit index and reads
at::cuda::getDeviceProperties(index), the SDK's own per-device cache,so the answer never depends on which device was selected earlier;
currentDeviceInfoisdeviceInfo (← getDevice). The test switches through every visible device in a fresh processand checks that
currentDeviceInforeports each index in turn.Clocks and bus width come from
cudaDeviceGetAttribute, since CUDA 13 removedclockRateandmemoryClockRatefromcudaDevicePropand the attribute enums exist in 12.x and 13.x. An indexat or above
device_countis aTORCH_CHECKfailure that surfaces as anIOerror, like everyother runtime control; the build without LibTorch fails the same way through
unavailable.crather than returning a record of zeroes. On a CPU-only SDK, where
cudaDevicePropdoes notexist, the exports compile under
TORCHLEAN_LIBTORCH_CUDAand fail throughIOthe same way. ACUDA-enabled SDK answers even while the bridge runs on the host: the queries describe the visible
devices, not the selected one.
What is not carried over
#30 also reported the architectures the binary held native code for (
__CUDA_ARCH_LIST__) andwarned when the device was being served through PTX JIT. Neither carries over, for two separate
reasons the module docstring keeps apart. The SDK's packaged architectures are a property of the
SDK build, and ATen's C++ API does not expose the list (only the Python package reads it back).
The kernels TorchLean compiles itself, the generated operations of
NN.Kernel, are compiled atrun time by NVRTC for the compute capability of the device they run on, so there is no
compiled-in architecture list for them to report.
Tests
NN/Tests/Runtime/Cuda/DeviceInfo.lean, called fromNN/Tests/Suite.leanunder every runtimestatus:
count, and memory;
currentDeviceInfoagrees withgetDevice; the index at the visible countis rejected; and a child process (
TORCHLEAN_LIBTORCH_DEVICE_PROBE=switch, the pattern of thememory probes) walks
setDevicethrough every device before any buffer exists, becausesetDevicerefuses while wrappers are live;setDevice; the switching probe requires a CUDA device (Buffer.requireCudaDevice, from feat(libtorch): run the bridge on the host when the SDK is CPU-only #36);deviceInfo 0fails.Checks run
RTX A4500 (sm_86, driver 580.126), pip torch 2.11.0+cu128 with a CUDA 12.8 toolkit for SDK
discovery and NVRTC, and pip torch 2.11.0+cpu with no toolkit; Lean v4.35.0, on
cfbcb9a5:Device-identity section of the GPU run:
On the host with the same binary the record is the same and the switching probe reads
skipped: the device switching probe needs the bridge on a CUDA device; on the CPU-only SDK,device identity: queries are rejected on a CPU-only SDK.Not run: the sanitizer harness and the elementwise C++ harness. This host has one GPU, so the
switching probe exercised one index; the multi-device case is covered by construction (per-index
reads) rather than by an observation.